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Abstract — In 1998, Zhang and Yeung found the first un- 
conditional non-Shannon-type information inequality. Recently, 
Dougherty, Freiling and Zeger gave six new unconditional non- 
Shannon-type information inequalities. This work generalizes 
their work and provides a method to systematically derive non- 
Shannon-type information inequalities. An application of this 
method reveals new 4-variable non-Shannon-type information 
inequalities. 

I. Introduction 

Let n be a positive integer, N the set {1,2,- • • ,n}, and 
V{N) its power set. Let £ = (£,i)i£N be an n-dimension 
discrete random vector. Let = (£i)iej, VI C N. For 
convenience, let £0 be a random variable taking a fixed value 
with probability 1. The entropy function of £ = (£,i)i£N 
maps sets I C N to the Shannon entropies H(^j), which 
take value on [0, +00]. An entropy function which takes finite 
values can be considered as a vector in the Euclidean space 
jjPCAO ff™' be the set of all such entropy functions. 
Therefore H™ 1 can be viewed as a region in R^W. It is 
known that the closure of -ff™', c/(Jf™'), is a convex cone 
[1]. A real function / on R^W is an information inequality 
if and only if f{x) > for all x G 

By Shannon-type information inequalities, entropy function 
has the following properties: is normalized, H^(%) = 0; 
nondecreasing, H^(I) < H^(J) for I C J C N; submodular, 
H^I) + J) >H^IUJ)+ H € (I n J) for I,JCN. Let 
Hn be the set of vectors in space satisfying the above 

three properties. Clearly, H m is a polyhedral cone. 

Obviously, C H N . It is known that H e 2 nt = H 2 , 

cl(Hf l ) = H 3 , and cl(Hf) ^ H N for n > 4 [1], [2]. 
Therefore, H n is an outer bound of cZ(iT^ ). In this work, 
all outer bound H™ ta referred will be taken for polyhedral 
cone. 

Since Shannon-type information inequalities can not fully 
characterize cZ(iJ™'), there must exist non-Shannon-type 
information inequalities. First unconditional linear non- 
Shannon-type information inequality was found in [2]. More 
of these non-Shannon-type information inequalities appeared 
in [3]-[6]. Very recently, Matus [17] derived an infinite se- 
quence of new 4-variable and 5-variable linear information 



inequalities, and used them to show that d(ff™') is not a 
polyhedral cone, for all n > 4. That is to say, cl(H e £ ) can not 
be fully characterized by a finite number of linear information 
inequalities. 

In the process of deriving non-Shannon-type information 
inequalities in [2] and [5], the authors used similar methods. 
This work generalizes their concept and method, then brings 
forward a method to systematically derive non-Shannon-type 
information inequalities. In particular, using this method, we 
find new 4-variable unconditional linear non-Shannon-type 
information inequalities. 

II. Polyhedral Cone and Its Projection 

In this section, we first briefly review the definition and 
properties of polyhedral cone, then turn to the projection of 
polyhedral cone. 

A polyhedral cone P in Euclidean space W 1 can be repre- 
sented in two ways: by the intersection of a finite number of 
closed half-spaces (an H-representation), 

P = {x G W 1 I Ax > 0}, where A e R rxn ; 

or by the nonnegative linear combination of a finite set of 
extreme rays (a V-representation), 



P = {x e 



Ry, y G M s , y > 0}, where R e 



Minkowski-Weyl Theorem states that these two representa- 
tions are equivalent [10]. 

In this work, a polyhedral cone is viewed as a region in 
Euclidean space as well as the finite set of linear inequalities 
representing the closed half-spaces defining the region. In this 
view, polyhedral cone H n can be considered as the region 
in space R p ( N ' > as well as a set of facet-defining Shannon- 
type information inequalities, namely, elemental forms of 
Shannon's information measures [8]. 

A projection of a region P in R" = R ni x I" 2 onto its 
subspace R" 1 is 



x\ G 



3^2 such that x = 



X-2 



GP 



That is to say, ir Xl is a projection determined by its range R ni 
and null space R™ 2 . And ir Xl (P) is the set of all the vectors 

ir Xl (x),x E P. 

The projection of a polyhedral cone is still a polyhedral 
cone. In its V-representation, projection of P onto R™ 1 is 
simply 

tt Xi (P) = {xi £ R" 1 I x x = R lV , yeR s ,y> 0}, 

where i?i e R™ lXS , i? 2 G R" 2XS , ^ ^ J = R. The 

projection of a polyhedral cone in its H-representation is more 
difficult. We will detail it later in Section [IV] 

Now we discuss some results relating a polyhedral cone 
with the half-spaces and vertexes of its projection. 

A homogeneous linear inequality ax > can be inferred 
from a system of homogeneous linear inequalities Ax > if 
and only if the result of the linear programming problem 

minimize ax 
subject to Ax > 

is greater than or equal to 0, if and only if there exists a vector 
y > so that y T A — a. The latter equivalence is in fact the 
well-known Farkas Lemma. Linear programming can be used 
either to verify the former linear optimization problem or to 
find a nonnegative solution of the latter linear equations. 

A set of homogeneous linear inequalities is called indepen- 
dent, if none of its inequalities can be inferred from other 
inequalities in the set. 

Lemma 1. A linear inequality ax\ > can be inferred from 
the projection of a set of linear inequalities A\X\ + A2X2 > 
onto R™ 1 ;/ and only if it can be directly inferred from 
A x x x +A 2 x 2 > 0. 

The lemma results from Projection Lemma used in block 
elimination algorithm [12], which is proved using Farkas 
Lemma. 

Lemma 2. A vertex x\ is in the projection of a set of linear 
inequalities A\X\ + A2X2 > B if and only if the result of the 
linear programming problem 

maximize (B — Aix±) T y 
subject to A\y = 0, 

y>0 

is less than or equal to 0. 

The equivalence can be readily proved by Farkas Lemma. 

Owing to the above two lemmas, with the knowledge of the 
extreme rays and linear inequalities of a polyhedral cone P\, 
one can verify whether it is the projection of polyhedral cone 
P, without actually computing the projection. 

III. Theoretical Background 

In this section, we present the theoretical background of the 
work. 

In 2™-dimension Euclidean space M. V ( N ' > , a vector is written 
as x v ( N ^> = {%i)i£-p{N)- I n tne discussion of linear informa- 
tion inequality, x$ is fixed to be 0. 



A. Projection of H n and H™' 

Definition 1. A projection of a region P in onto its sub- 

space M?^' is tt x -p(m) (P), where M — {1, 2, • • • , m}, m < 
n. 

This is the same as the concept of restriction in [6], [17]. 

Proposition 1. ir w v lM )(H%) = Hfj. tt x v { m ) {cI(H^')) = 
cl{HZ). 

The proposition can be easily proved by truncating an n- 
dimension discrete random vector. 

Proposition 2. n x p(M) (H N ) = H M - 

B. Procedure for Derivation of Non-Shannon-Type Informa- 
tion Inequalities 

Following, we state the procedure for systematically deriv- 
ing non-Shannon-type information inequalities. Below, we re- 
strict consideration to random vectors whose entropy functions 
take finite values. 

Say, we want to find m- variable information inequalities. 
For any m-dimension random vector £, we construct a corre- 
sponding n-dimension (n > m) random vector £ = (Q)ieN, 
so that the marginal distribution of its first m random variables, 
namely, the probability distribution of equals that of £. 
Let Aa? in R 73 ^' be the set of entropy functions of all these 
n-dimension discrete random vectors. Observing the above 
construction and recalling the definition of projection, it fol- 
lows that h x v{m) (Aat) = i?™'. Therefore, if we can to some 
extent characterize A^v, thus its projection h x v{m) (Ajv), we 
can finally arrive at an outer bound of H^. Obviously, H n 
is a trivial outer bound of Ajv- Since tt x t(m)(H m) = Hm, 
it provides no new information about H e ^. However, for 
properly designed Q, some better outer bound of An can be 
reached. 

In the latter part of the section, we focus on the technique 
used in [5] to construct the n-dimension random vector £ and 
the outer bound of A at. 

We restate a lemma in [9, Lemma 14.8]. 

Lemma 3. Given an n-dimension random vector £ = (^i)igjv, 
there exists a random variable such that: 

1) The joint probability distribution of (£ n +i,£/) equals 
that of^k,^), 

2) Markov chain £,{k}uJ ~ > ~* £n+l holds, 
where {fc},/, J are disjoint subsets of N. 

In [5], random variable £„+i is called a £j-copy of over 
£/■ 

Therefore, with £ n+ i, f = (£1, • • • , £nj£n+i) is an n + 1- 
dimension random vector. And 

H c ({n + 1} U h) = H c ({k} U h), h C I, 

H ( ({n+l}{Jl)+H ( {PJJi) = H ( {{n+l}UlU JJ + H^I), 

Ji C {k} U J, Ji ^0, 

for all C 



For constructing an n-dimension random vector from m- 
dimension random vector £, we start with £, then add auxiliary 
random variables £ m +i, • • • ,£ n according to Lemma[3] choos- 
ing parameters k, I, J every time. The process ends with an 
n-dimension random vector C = (£1, • • • , £ m , Cm+i, ' • • , £«)• 
Besides all n-variable Shannon-type information inequalities, 
depending on the parameters chosen, the entropy function 
of £ also satisfies the above two kinds of equalities. Let 
Cat denote the set of all these additional linear equalities. 
The region of the entropy functions of all £, A^r, is outer 
bounded by the polyhedral cone H ^ n Cjy. Recalling that 
Hm = n x vw)(A N ), it follows that ff$ C d(H%) C 
w x v(m)(Hn n Cm)- The projection of H Cat can be 
explicitly calculated. If 7T a .'P(M) (iTjv H Cat) is strictly smaller 
than Hm, we then have a better characterization of H™}. 
In other words, some of the facet-defining linear inequalities 
of ir x -p(M) (Hn ("I Cat) must be m-variable non-Shannon-type 
information inequalities. 

In the above procedure, alternatively, we could use any 
known outer bound H™ teT instead of H m- That is to say, we 
calculate the projection of H N tel fl Cjy instead. This means 
using known linear non-Shannon-type information inequali- 
ties in the derivation of new non-Shannon-type information 
inequalities. 

We delay the application of above procedure to Section [V] 

IV. Projection Algorithm 

Although projection of a polyhedral cone in its V- 
representation is simply the projection of all its extreme rays, 
in this work, we prefer the H-representation, partly because 
we are interested in linear information inequalities, and partly 
because the V-representation of an outer bound might be 
exponentially more complex than its H-representation. 

In order to derive non-Shannon-type information inequal- 
ities using the theory and method mentioned in Section [Hi] 
we must actually calculate the projection of a polyhedral 
cone. However, even for deriving 4-variable non-Shannon- 
type information inequalities by projecting the polyhedral cone 
outer bounding the entropy functions of all corresponding 6- 
dimension random vectors, it requires calculating the projec- 
tion of a 2 6 -dimension polyhedral cone consisting of hundreds 
of linear inequalities onto a 2 4 -dimension space. Moreover, 
there is no explicit complexity results known for polyhedron 
projection [12]. 

Classical projection methods such as Fourier-Motzkin elim- 
ination and block elimination are too computationally ineffi- 
cient to project high dimension polyhedron [11], [12]. And the 
degeneracy of the polyhedral cone may render other projection 
algorithms, such as ESP [15], inefficient. 

We turn to convex hull method (CHM) proposed in [13]. It 
works directly in the projection space. The flowchart of CHM 
is depicted in Fig.Q] For projecting a polyhedral cone P in M™ 
onto R™ 1 , the algorithm incrementally constructs a polyhedral 
cone Pi in W 11 while maintaining its double description pair 

J , R\ j . The output is n Xl (P) = Pi, represented by 



Input: 
Facets A of P 



I 



Construct an initial 
P\ with facets A i 
and extreme rays R\ 




Output: 
Facets A2 and extreme ] 
rays Ri of k(P) 



Choose a facet a 
from A 1 




Move a outward to 
find extreme ray r 



Add r to fi 1 
Update P h A , by.fi, 
using convex hull 
agorithm 



Fig. 1. Flowchart of convex hull method for projection 



double description pair (A2, Ri). 

The convex hull algorithm used in the CHM can be imple- 
mented using any incremental algorithm. We adopt Fourier- 
Motzkin elimination (the dual of double description method 
[14]), since it appears to deal well with degeneracy [16]. 

We can take advantage of the incremental nature of CHM 
to further reduce its computational complexity. Let J?™ er be 
an inner bound of cl(H™j). Then the extreme rays of H m in 
.ff 1 ™ 61 must be the extreme rays of cl(H e ^), as well as the 
extreme rays of any TT x v{M)(H°^ el n CV). If the polyhedral 
cone generated by these extreme rays is full dimensional 
(except on axis x$), we can then skip the initialization and 
immediately start CHM from this approximation. This method 
can be used for projection onto R'Ptt 1 ' 2 * 3 ' 4 }), since the inner 
bound of cl{HT) is known [2], [7]. 



V. New 4-variable Non-Shannon-Type Information 
Inequalities 

In this section, we focus on 4-variable non-Shannon-type 
information inequalities. They are of theoretical importance, 



meanwhile, low dimension polyhedral cones are more compu- 
tationally tractable. In the following, we discuss two different 
approaches for calculating new non-Shannon-type information 
inequalities using the procedure introduced in Section IIII-BI 
Then the results are used to help derive an infinite sequence 
of 4-variable non-Shannon-type information inequalities. 

A. Projection by Adding More Random Variables 

First we add the 5th and 6th random variables, as what is 
shown in the proof of [5, Theorem III. 1 ] . For any probability 
distribution of 4-dimension random vector £, by Lemma [3] 
let £5 be a £ 4 -copy of £3 over £{i,2}> & be a £2-copy of 
£3 over £{1,4,5}. This results in a 6-dimension random vector 
C = , ^6) ■ And the corresponding polyhedral cone Cq 

has 18 equalities. tt x -p(u,2,3a}) {He DCq) is a polyhedral cone 
defined by 35 linear information inequalities, which reveals 
Zhang-Yeung inequality and the third and fifth Dougherty- 
Freiling-Zeger inequalities. 

We use more than 6 random variables. For instance, let 
£7 be a £ 4 -copy of £ 2 over £{1,3,5,6}- And C = (£1, • • ■ ,£7)- 
The corresponding polyhedral cone C7 has 19 more equal- 
ities. The resulting projection it x -p((i,2,3a}){H 7 n C 7 ) is a 
polyhedral cone defined by 56 linear information inequalities, 
which reveals 13 independent non-Shannon-type information 
inequalities unknown before. Due to space limitations we just 
list several of them. 

- 56#(£ {1} ) - 4if(£ {2} ) - 19H(£{3 } ) 

+ 45 J ff(£ {1 , 2} ) + 67ff (£ {1 , 3} ) + 22 J ff(£ {2 , 3} ) 
+ 23ff(£ {M} ) - 8ff(£ {2 . 4} ) + 9tf(£{ 3 ,4}) 

- 55ff(£ {1 , 2> 3}) ~ 24ff(£ {1>3 ,4 } ) > 0, 

- 34ff(£ {1} ) - 2ff(£ {2} ) - llff(£{ 3 }) - # (£ {4} ) 
+ 27if(£ {1>2} ) + 40#(£ {1 , 3} ) + UH (£ {2 , 3} ) 

+ 15ff(£ {1>4} ) - 5tf(£ {2 , 4} ) + 7H(£ {3A} ) 

- 32H (£{1,2,3}) - 16if(£ {1>3 ,4}) > 0, 

- 28ff(£ {1} ) - ff(£ {2} ) - 10ff(£ {3} ) - 2ff(£ {4} ) 
+ 22H(t {h2} ) + 34ff (£ {1 , 3} ) + im(£ {2 ,3 } ) 

+ 13ff(£{ 1)4} ) - 4tf(£ {2 , 4} ) + 6ff(£ {3 , 4} ) 

- 28ff(£{ 1)2>3} ) - 13ff(£ {1>3) 4}) > 0. 

The novelty and independence of these information inequal- 
ities can be verified by finding an extreme ray which does 
not satisfy one of the inequalities, but satisfies all 4-variable 
Shannon-type information inequalities together with all sub- 
stituted forms of Zhang-Yeung inequality, Dougherty-Freiling- 
Zeger inequalities, Matus inequalities, and the rest of these 
inequalities. The verification can be done by linear program- 
ming. 

Nevertheless, without calculation (projection or verifica- 
tion), we do not know beforehand how these auxiliary random 
variables would affect the resulting projection. This hinders us 
from systematically designing the random vector £. 



B. Projection Using Improved Outer Bound 

Alternatively, we choose not to add that many auxiliary 
random variables, but to use an outer bound strictly smaller 
than Hm- 

Let £5 be a £ 4 -copy of £3 over £{i )2 }. During process in 
Section [V-AI it is already known that 7T x -p({i,2,3,4}) (H$ D C5) 
only reveals Zhang-Yeung non-Shannon-type information in- 
equality, 

-2ff(£ {1} )-2i/(£ {2} )-iJ(£ {3} ) 

+ 3ff(£ { i >2} ) + 3ff(£ {1 , 3} ) + 3ff(£ {2 , 3} ) 

+ ^(£{l,4})+^(£{2,4})-^(£{3,4}) 

- 4ff (£{1,2,3}) - #(£{1,2,4}) > 0. (1) 

An outer bound jj™"^ 1 ) j s constructed from H4 and Zhang- 
Yeung inequality. 

Through substitution, an outer bound jf° uter ( 1 ) can jj e 
constructed from jf° uter ( 1 ) i w e repeat the procedure, but 
project jf™ 11 ^ 1 ) p| q s rather than H§ n C5. The resulting 

is a polyhedral cone defined by 
55 linear information inequalities, 6 of which are indepen- 
dent information inequalities that can not be inferred from 
.ffou . Among them, the following is unknown before, 

-10iJ(£ {1 })-10if(£{ 2 })-if(£{ 3 }) 

+ 17ff(£{l,2}) + lOi? (£{1,3}) + 10#(£{2,3}) 

+ 4 J ff(£ { i,4}) + 4# (£ {2 , 4} ) - 3iJ(£ {3 , 4} ) 

- 16ff(£ {1 , 2>3 }) - 5iJ(£ {1 , 2 , 4} ) > 0. (2) 

Outer bound _ff° uter ( 2 ) [ s constructed from jf° uter W together 
with these new information inequalities. 

It can be observed that the above procedure is indeed 
a function which maps if 4 to ff^ utor , and subsequently 

j^outer(l) tQ j^outer(2) 

C. Infinite Sequence of 4-variable Non-Shannon-Type Infor- 
mation Inequalities 

Though the projection algorithm will only reveal finite 
number of non-Shannon-type linear information inequalities, 
the information acquired from the results may still help shed 
some light on the infinite sequence of linear information 
inequalities. 

Once a new m-variable linear information inequality is 
derived by projecting some if™ ter n Cm, it can be written 
as a nonnegative linear combination of those n-variable in- 
formation inequalities and additional equalities, as discussed 
in Section HI] This nonnegative linear combination can be 
viewed as an explicit proof of this newly derived information 
inequality. 

It can be observed that Zhang-Yeung information inequality 
(Q3 and information inequality (fJJ bear some similarity. More- 
over, the explicit proofs corresponding to the two information 
inequalities share a similar structure. Therefore, with several 
more iterations and some guesswork, we derive the following 



infinite sequence of 4-variable non-Shannon-type information 
inequalities. 



> n' . Therefore, we define H 



nTT' 
n>m 



MJ.JV' 



And 



obviously H°^ N D flj, Then, the critical issue is whether 



V2 „ V2 



1 - 3 • 2 s 



2 ^ 



ff(£ { i, a} ) 



tt outer 



+ Qs + + [fr(e {1 ,3 } ) + J ff(f {2 ,3})] 

+ (^rV-^±V) [^K{i.4 } )+ ff(f {2 , 4} )] 

+ (1 - 2 s - 1 ) 77(£ {3)4 }) + (2 s - 1 - - i,S_) 77(^,2,3}) 

- (l - + - ^5T_) H« {1A 4}) > 0' 

where 5+ = (2 + v^V , S- = (2 - %/2) S , and s e N. (3) 

A rigorous proof can be easily obtained through mathematical 
induction. 

When s = 1, information inequality (0 corresponds to 
a Shannon-type information inequality. For s = 2 and 3, 
it is Zhang- Yeung inequality and information inequality (O, 
respectively. 

VI. Discussion and Future Work 

The concept and usage of projection is not limited to the 
theory and application mentioned above. The notion of infer- 
ence rule in [3] and inner adhesivity in [17] and their methods 
for deriving non-Shannon-type information inequalities can 
also be regarded as a practice of projection. For example, 
in the proof of the 5-variable non-Shannon-type information 
inequality in [3], the application of the inference rule is 
equivalently the following set of additional equalities, 

^{3,4,5} + £{3,4}UJ - £{3,4,5}UJ ~ ^{3,4} = 0, 

JC{1,2},J^0, 

and the requirement that no term in the resulting inequality 
has any of {1,2} together with {5} specifies the range of 
the projection, i.e. the subspace consisting of all vectors with 
sjuju{6} = 0, 7 C {3, 4}, J C {1, 2}, J ? 0. 

The procedure used in Section IV-AI can be generalized and 
its limit property is to be investigated. Let H°^ N denote the 
intersection of all projections tt x v(m) (HnHCn) derived using 
the procedure described in Section IIII-BI applying Lemma [3] 
For example, it can be shown that ffjffg is in fact the region 
enclosed by all 4-variable Shannon-type information inequal- 
ities and all substituted forms of Zhang- Yeung inequality, and 
that H™^ 1 is the region enclosed by all 4-variable Shannon- 
type information inequalities and all substituted forms of 
Zhang-Yeung inequality and Dougherty-Freiling-Zeger in- 



cl(H™j). If the above equation holds, then this 
work presents an explicit characterization of d(ff™/), and we 
are capable of calculating and approximating it. Otherwise, 
the problem is whether there exist other methods of adding 
random variables and other kinds of additional inequalities 
that can be used as Cat, so that after a similar process the 
newly obtained i7^' N would coincide with cl(H™\). 

Similarly, the procedure described in Section IV-BI can be 
generalized. Let a be a function defined on the set of polyhe- 
dral cones in R V ( M \ which maps an outer bound H°^f el to the 
intersection of all projections ir x v(,M){H°if st n Cjv), for fixed 
n. Clearly, a k+1 (H M ) C a k (H M ), and a k (H M ) 2 HjJ- 
Then, the problem is whether er fc+1 (i7^/) is strictly smaller 
than o k (H M ), and whether limfc_ >00 a- k (H M ) = cl(H™j). 
These call for further investigation. 
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equalities. It is easy to see that H 



outer 

MJ.JV 



C H 



MJ.JV 



, for 



